Skip to main content

View Post [edit]

Poster: bibble01 Date: Jan 1, 2012 2:11am
Forum: web Subject: How often does archive crawl a specific website?

Hi,

I am new to internet archive. I am curious how often the archive crawls a specific website. Look like it varies on different websites.

thanks,

Reply [edit]

Poster: Jeff Kaplan Date: Jan 2, 2012 4:56pm
Forum: web Subject: Re: How often does archive crawl a specific website?

there is no specific schedule. if it is well linked it will have a higher likelihood of being crawled more often.

Reply [edit]

Poster: bibble01 Date: Jan 3, 2012 6:24pm
Forum: web Subject: Re: How often does archive crawl a specific website?

Jeff, thanks for your reply. So even some news website constantly update the page, it doesn't mean that the crawler will catch the update, right?

Another question: why does it take 6 months to transfer the storage? It is a little long...

thanks,